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REAL PARTY IN INTEREST 

6S, Ltd. is the real party in interest for all issues related to this application. 6S, Ltd. 

owns this patent application by virtue of an assignment recorded with the Office at reel 011480, 
frame 0535. 
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RELATED APPEALS OR INTERFERENCES 

There are no other appeals or interferences related to this application. 
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STATUS OF CLAIMS 

This application contains claims 1-27 and 58-67, all of which stand rejected as obvious 
over prior art. All rejections are appealed. 
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STATUS OF AMENDMENTS 

No amendments have been filed subsequent to the final rejection in this application. 
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SUMMARY OF CLAIMED SUBJECT MATTER 

The present invention provides methods and systems to tailor indexing and retrieval of 

digital media items based on episodic memory of predefined groups of people. A system 
implementing aspects of the invention will generally be directed at one or more predefined 
groups of people, such as families, work groups, and sports teams. When a user operates the 
system, he is identified as a member of a group. The user may then archive digital media items 
based on event types, other users in the group and/or dates and times. These categories and 
attributes are predefined and associated with the group to which the user belongs. Similarly, 
when a user wishes to retrieve digital media items from the system, he identifies himself and is 
recognized by the system as being a member of a particular group. The user may then retrieve 
digital media items based on criteria that are specific to that group. 

The invention also allows for digital media items to be stored and retrieved as "high 
points" - that is, digital media items that represent or embody a particularly memorable 
moment for the group. Users may also define "trails" of digital media items, where a set of 
media items are arranged in a logical or meaningful order as perceived by a member of the 
group. Trails can be retrieved and displayed automatically, i.e., after a user selects one of the 
elements of the trail, or manually, when a user follows one media item in the trail to the next. 
The use of trails and high points enhances the nostalgic experience provided by the episodic 
archival and retrieval systems. Compared to semantic retrieval systems, i.e. those where an 
item is identified by describing facts about it or concepts related to it, an episodic archival 
system is based on remembered events. Thus it provides many benefits in recording the 
common memories of members of a group. Instead of archiving and retrieving media items 
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based on their content, users may do so based on the mutual significance that each media item 
has to the group. 

Archival of Media Items bv Group Members 

First, a group of users 1 is registered 2 
with the system. The group is stored in a 
database 3. Members of the group may then 
work together to archive digital media items, 
such as audio recordings, digital photographs, 
digital video clips, and the like. When a user 
wishes to use the system, he first provides 
identifying information such as a name and 
password; this information identifies him as a 
member of a group. The user can then import 
one or more digital media items 6 into a 
workspace. The system generates identifiers 
that can be associated with each item. For example, the system may provide lists of people 3, 
event types 4, and/or dates 5 with which each digital media item may be associated. For each 
digital media item, the user can choose information from each list to associate with the item. 
Index information is created for each combination of digital media item and chosen information 
in a storage process 7, and the item is then stored in a multimedia database 8. When a user 
later wishes to retrieve a digital media item, he may do so by specifying one or more users, 
types of events, and/or dates and times 9. See Figs. 6-7 and pp. 12-14 of the application. 




Fig 1 
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Users may also indicate "high points" (media items that are especially significant) and "trails" 
(a sequence of media items arranged in a particular order). 

The information available for a user to associate with a media item is determined based 
on the user's membership in a group, as defined in the group registration process 2. A user is 
only presented with options relevant to the group of which he is a member. Thus, a group of 
users can utilize the system to record media items related to event types, people, and dates 
that are common to members of that group. 

Claims 1, 58, and 66 capture the archival of digital media items. Claim 1 recites that, 
when user input identifying a group to which a user belongs and data identifying information 
about a digital media item are received, index information based on the input is created. The 
indexing information is then stored in association with the media item. Similarly, claim 58 
recites that, when an operator is identified as a member of a group of users, candidate 
information values based on the group to which the operator belongs are identified. The user is 
then queried for a selection of identification data to be associated with a digital media item. 
Index information is created based on the operator's response, and stored in association with 
the digital media item. Claim 66 recites that a database is built when user input identifying a 
social group to which the user belongs is received. The database is comprised of the digital 
media items provided by the user and index information created from archiving input data 
received from the user. 
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Retrieval of Media Items bv Group Members 

After digital media items have been archived in the system, users may wish to retrieve a 
selection of the archived items. To do so, a user first presents identifying information to the 
system and selects retrieval parameters 90. As described above with reference to the archival 
process, the types of retrieval parameters presented to a user may be determined from the 
group to which the user belongs. The user may choose to retrieve media items that are 
designated as "high points," to follow predetermined "trails," or to retrieve items associated 
with one or more people, time periods, and/or event types. See pp. 14-15 of the application. 
Indexing information based on the parameters selected by the user is created, and items 
matching the indexing information are identified 91 and presented to the user 92. Other 
navigation options may be presented to 
the user, such as following a trail if one 
or more of the presented media items is 
a trail item 93-94. 

Claims 1 and 62 capture the 
retrieval of digital media items. Claim 67 
captures methods for searching and 
retrieving archived media items. Claim 1 
recites that digital media items are 
retrieved based on input data representing a selection of group event types, one or more 
people in the user's group, and/or a time period. Claim 62 similarly recites a method of 
authenticating an operator as a member of a group and providing information for selection by 
the operator based on the operator's group membership. The operator is then queried for 



USER SELECTS RETRIEVAL | 
PARAMETERS 




607412_1.DOC 



- 10- 



13642/1 



Appeal Brief 
Appln. No. 09/670,635 
Atty. Dkt. No. 13642/1 

selection of identification data that includes selections of event types, persons, and/or time 
periods. Index information is generated from this data, and stored media items corresponding 
to the index information are retrieved. Similarly, claim 67 recites identifying candidate 
information values based on a social group of which a user has identified himself as a member. 
A query is then displayed, which identifies the candidate information values. Based on a user's 
selection of criteria such as even types, persons from the group, and/or time periods, a 
database is searched for digital media items satisfying the selection criteria. The digital media 
items are then retrieved and may be presented to the user. 
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GROUNDS OF REJECTION TO BE REVIEWED ON APPEAL 

1. Whether the outstanding §103 rejections to claims 1-27 and 66 over Shneiderman et 
al., "Direct Annotation: A Drag-and-Drop Strategy for Labeling Photos," July 2000 
("Shneiderman") in view of E.P. 0 678 816 A2 to Mizoguchi ("Mizoguchi") should be 
reversed. 

2. Whether the outstanding §103 rejections to claims 58-65 over Shneiderman in view of 
U.S.P. 5,485,611 to Astle ("Astle") should be reversed. 

3. Whether the outstanding §103 rejections to claim 67 over Mizoauchi in view of 
Shneiderman should be reversed. 
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GROUPING OF CLAIMS 

The claims do not stand or fall together. The argument provides a detailed discussion of 
the elements of the various claims. 
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ARGUMENT 

The obviousness rejections to claims 1-27 and 58-67 must be reversed because the 
references, even when taken collectively, fail to teach or suggest the subject matter of the 
pending claims. 

An obviousness rejection must satisfy three basic criteria. First, there must be some 
suggestion or motivation, either in the references themselves or in the knowledge generally 
available to one of ordinary skill in the art, to modify the reference or to combine reference 
teachings. Second, there must be a reasonable expectation of success. Finally, the prior art 
reference (or references when combined) must teach or suggest all the claim limitations. The 
teaching or suggestion to make the claimed combination and the reasonable expectation of 
success must both be found in the prior art, and not based on applicant's disclosure. In re 
Vaeck, 947 F.2d 488, 20 USPQ 2d 1438 (Fed. Cir. 1991); MPEP § 2142. Because the cited art in 
this application simply does not teach or suggest all claim limitations, the appealed rejections 
must be reversed. 

The Examiner's primary reference, Shneiderman, discloses a system for labeling 
photographs in which a user can select a label to be placed on a digital photograph from a list, 
and drag the label onto the photograph. A name only has to be typed in the first time it 
appears in a photograph. A user interface to the system described in Shneiderman is shown in 
Fig. 3. It includes areas in which a user can inter a title, description, and keywords; start and 
end dates; and a location describing the current photograph (bottom left). A user can also type 
in a name with which to annotate the photograph, or select a previously-used name from a 
master list (bottom center). See Shneiderman, p. 4, Figure 4; p. 6, col. 2. The photographs 
being annotated are stored in a file on the computer ("C:\My Photos\samplehcilibrary.mdb" in 
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the title bar). Id., Fig. 4; p. 7. Notably, the entire list of people names stored in the application 
is presented each time an operator wishes to annotate a photograph. Id., p. 5, col. 1. There is 
no indication or suggestion that the list of names available as annotations is linked to or 
dependent on the operator of the program. The list of names is stored in a database table, and 
there is no indication in Shneiderman that the names are associated with groups. See id, p. 7, 
col. 1; Figure 6. 




Shneiderman's FIG. 4. 



Mizoguchi discloses a method of generating image data which allows for searching the 
image data at a high speed. Mizoguchi , abstract. The time at which a photograph is taken with 
a digital camera is compared with a predetermined time range; information corresponding to 
the predetermined time range is stored with the image. Id. A schedule of events is stored in 
the camera. When a picture is taken with the camera, any data associated with the time the 
picture is taken is then used to index the picture. See id, col. 7, lines 26-42. The methods and 
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systems described in Mizoquchi rely on the camera having electronic notebook, scheduling, and 
data management functions. See id, col. 7, lines 1-5. For example, a person may schedule a 
golf game with three other people, for a specific date and time, in the scheduling component of 
the camera. When a photograph is taken during the golf game, the photograph is associated 
with the event f golf") and the date, time, and other people listed on the schedule. See id, col. 
8, lines 26-42. Stored photographs may then be searched by this associated data. See id, col. 
10, lines 34-47. 

Astle describes a method for generating a video database index, where video sequences 
are indexed based on index frames. See Astle, abstract; col. 6, lines 9-11. An index frame is 
selected based on the amount of change in scenes depicted in the video database; each index 
frame represents a video sequence. Id, col. 6, lines 4-11. In the background section, Astle 
describes indexing a video sequence by denoting the location of the sequence on a particular 
video cassette by a time index or a counter index. Id, col. 2, lines 11-25. 

None of the cited references disclose any system for leveraging the episodic memory of 
a social group of people. That is, the references fail to disclose or suggest a retrieval system 
that permits nostalgic retrieval among a broad set of digital media items - beyond merely 
photographic items - or permits operators to browse among such sets of items. When 
compared against the requirements of the pending claims, it becomes clear that the claims 
define patentable inventions over this art. 
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Rejections under 35 U.S.C. S 103fa) over Shneiderman in view of Astle 

Independent Claims 58 and 62 

Claim 58 recites a media archival method; claim 62 recites a related media retrieval 

method. These claims stand rejected as obvious over Shneiderman in view of Astle . Both 
claims read, in pertinent part: 

authenticating an operator as a member of a group of users, 

identifying candidate identification values based upon the group with whom 
the operator is authenticated, 

querying the operator for selection of identification data to be associated with a 
digital media item, the query identifying the candidate identification 
values and including valid selections of an event type and persons from 
the group and time ... 

None of the cited references teach or suggest this subject matter. None of the references 
teaches or suggests the step of authenticating an operator as a member of a group of users, 
identifying candidate identification values based on the group or providing a query that includes 
valid selections of an event type and persons from the group. Accordingly, the rejections to 
claims 58 and 62 must be reversed. 

The Examiner argued that Shneiderman, in FIG. 4, teaches the authentication step in 
FIG. 4. In both the Dec. 12, 2005 and Jan. 26, 2006 Office Actions, the Examiner argued that 
the reference to C:\My Photos\samplehcilibrary.mdb demonstrates a log on process that 
corresponds to the authentication step. The Examiner is wrong. Shneiderman 's reference 
merely identifies a location of an mdb database file in an ordinary computer file system. Claims 
58 and 62 specify that the authentication step authenticates an operator as a member of a 
group of users, then further uses this group-based information to govern further operation of 
the respective methods. Shneiderman 's disclosure is not so specific as to describe any process 
that authenticates an operator as a member of a group of users. None of the supplementary 
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art is cited for this disclosure. This is a first basis on which to reverse the Examiner's rejection 
to claims 58 and 62. 

Additionally, Shneiderman fails to teach or suggest identifying candidate identification 
values based on the group with whom the operator is authenticated or querying an 
operator including valid selections of an event type and persons from the group. These 
features, which correspond to the principles of episodic memory described in the specification, 
are lacking from any reference. Shneiderman allows an operator to enter names and 
description information simple as freestyle text. For convenience, an operator may drag-and- 
drop previously entered names from a pull down menu onto a photograph. See Shneiderman, 
p. 5, col. 1. Shneiderman has no disclosure, however, to support an inference that his system 
poses queries to an operator using identification values that are based on the group or it 
includes within those queries valid selections of an event type or persons from the group. 
Freestyle text entries and drag-and-drop operations are insufficient to meet the substance of 
these claim elements. None of the supplementary art is cited for this disclosure. This is a 
second basis on which to reverse the Examiner's rejection to claims 58 and 62. 

Applicants clearly explained the deficiencies of the cited art during prosecution. See 
Response of July 15, 2005; Response of Feb. 10, 2006. The Examiner never directly addressed 
the arguments, but in reply merely repeated verbatim the previous rejections. See Office 
Action, p. 7-9 (Jan. 26, 2005); Office Action, p. 9-10 (Dec. 12, 2005). Of course he cannot, 
because the art does not teach or suggest this subject matter. The cited art does not teach the 
principle of archival based on episodic memory nor does it teach the claim elements cited 
above. 
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Dependent Claims 59 and 63 

Claims 59 and 63 depend respectively from claims 58 and 62 and further recite that the 

candidate identification values for persons include names of group members. The Examiner 
relied on seemingly random disclosures of Shneiderman in making obviousness rejections of 
these claims. The Examiner asserted that the use of "names of group members" in claims 59, 
and 63 was rendered obvious by Shneiderman 's disclosure that a user may select names in the 
library. It is unclear what relationship the Examiner sees between the use of group member 
names and photograph annotations. The group member names of claims 59 and 63 refer to 
groups of users of the archiving system. In contrast, the person names and descriptions 
disclosed in Shneiderman are annotations referring to the contents of the annotated 
photographs. There is no relationship between the names in the master list and the operator of 
the program in Shneiderman . The art simply is not relevant to the archival process recited in 
the claims. The rejection of claims 59 and 63 is incorrect and should be reversed. 

As with claims 58 and 62, Applicants have explained these differences to the Examiner, 
but Applicants' arguments have never been addressed. 

Dependent Claims 60 and 64 

Claim 60 depends from claim 58; claim 64 depends from claim 62. Both claims recite 

the additional feature that the stored index information includes a flag that distinguishes high 
point items from other items. The Examiner claimed that the use of a high point flag in claims 
60 and 64 1 was rendered obvious by Shneiderman 's disclosure of "show name labels." Office 



1 In the Office Action of Dec. 12, 2005, the Examiner identified claims 61 and 65 as relating to 

setting a high point. Claims 61 and 65 do not recite a high point; rather, they are directed to the use of a trail. 
Presumably, this rejection was meant to apply to claims 60 and 64. It is unclear to what the rejection given for 
claims 60 and 64 is directed. Regardless, the cited art is irrelevant to the claims. 
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Action of Dec. 12, 2005, p. 10. The fact that annotations may be shown on a photograph in 
Shneiderman has no relation to the use of the high point flag recited in claims 60 and 64. As 
clearly described in the specification, the high point flag allows archiving users to mark a digital 
media item as related to a particularly memorable event. An example of such an event is a 
group's last day of school. See application, p. 12. In contrast, Shneiderman merely refers to 
displaying names of people in a photograph. The disclosure of Shneiderman is simply not 
relevant, and the rejection of claims 60 and 64 is incorrect and should be reversed. 

As with claims 58 and 62, Applicants have explained these differences to the Examiner, 
but Applicants' arguments have never been addressed. 

Dependent Claims 61 and 65 

Claim 61 depends from claim 58; claim 65 depends from claim 62. Both claims recite 

the additional feature that if a digital media item is a member of a trail, the index information 
includes an identifier representing the media item's display position in a sequence of stored 
media items. None of the cited art teaches or suggests that use of a "trail" as recited in the 
claims and described in the specification. The Examiner has never given a satisfactory reason 
for rejecting these claims. In two consecutive Office Actions, the Examiner rejected both claims 
on the reasoning the Shneiderman discloses the use of a "high point." Office Action of Jan. 26, 
2005, p. 9; Office Action of Dec. 12, p. 10; see footnote 1, above. However, claims 61 and 65 
recite the use of a trail, not a high point. The rejection of claims 61 and 65 is baseless, and 
should be reversed. 

As with claims 58 and 62, Applicants have explained these differences to the Examiner, 
but Applicants' arguments have never been addressed. 
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Rejection under 35 U.S.C. S 103(a) over Mizoauchi in view of Shneiderman: 
Independent Claim 67 

Claim 67 was rejected as obvious over Mizoguchi in view of Shneiderman . It recites: 
A method of searching digital media items, comprising: 

receiving a user input identifying a social group for which a search is to be 
conducted; 

identifying candidate identification values based upon the social group, 

displaying a query that identifies the candidate identification values and including 
valid selections of an event type for the social group, persons from the 
social group and time, 

responsive to selection criteria made in response to the query, searching a 
database and retrieving digital media items that satisfy the selection criteria. 

As previously explained, Shneiderman fails to teach or suggest identification of social groups, 
identifying candidate identification values based on the group or displaying a query that 
includes valid selections of an event type for the group. The Examiner admitted that Mizoguchi 
does not disclose identifying candidate identification values based on a social group. Office 
Action of Dec. 12, 2005, p. 8. Therefore, the rejection should be reversed. 



Rejection under 35 U.S.C. S 103fal over Shneiderman in view of Mizoauchi 
Claim 66 

Claim 66 was rejected as obvious over Shneiderman in view of Mizoguchi . It reads: 
A method of archiving digital media items, comprising: 

receiving a user input identifying a social group to which an archiving 
user belongs, 

building a database that includes: 

digital media items to be archived for the social group, and 

index information for the digital media items, each instance of index 
information created from archiving input data identifying a user's response to a 
query that identifies a plurality of event types previously registered as 
associated with the social group, and persons previously registered as 
members of the social group. 
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The cited art fails to teach or suggest all elements of this claim. No reference, for example, 
suggests that index information is derived from a query that identifies event types that have 
been previously registered as associated with a social group. Although Shneiderman refers to 
event information generally, he has no disclosure to suggest that event information is selected 
from a query that identifies event types. Shneiderman certainly does not disclose that 
these event types are previously registered in association with a social group. 
Mizoauchi discloses no use of social groups or event types associated with social groups. The 
cited art fails to teach or suggest all elements of claim 66 even when considered collectively 
and, therefore, this rejection must be reversed. 



This argument also was presented to the Examiner in the Response to Final Office 
Action of February 16 and left unrebutted. The Examiner merely asserted that the arguments 
were not persuasive. The Examiner's indifference to these arguments is surprising, given the 
relevance these features hold to the episodic memory principles that underlie the pending 
claims. 



Claim 1 

Claim 1 stands rejected as obvious over Shneiderman in view of Mizoauchi . It reads: 

A method of archiving and retrieving digital media items, comprising: 

receiving a user input identifying a group of users to which an archiving 

user belongs; 

receiving archiving input data identifying: a digital media item to be archived for 
the group, the user's selection of zero or more group event types from a 
predetermined plurality of group event types specific to the group, the 
user's selection of zero or more persons in the group, and the user's 
selection of a time period; 

• • • 

receiving retrieval input data representing a selection of a default or zero or 
more group event types from the predetermined plurality of group 
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event types for the group, a selection of a default or zero or more 
persons in the group, and a selection of a time period; and 

using the selections and the identified group to retrieve and output digital 
media items that match the selection. 

The Examiner concluded that Shneiderman teaches the first five steps, and that Mizoquchi 
provided a way to modify the steps alleged to be taught in Shneiderman to include the last two 
steps. The Examiner did not demonstrate a motivation in either reference to combine the 
teachings of Mizoquchi with the teachings of Shneiderman . During an interview held February 
9, 2006, the Examiner suggested that the references' failure to describe every element of the 
claims is a minor omission. 

The Examiner incorrectly interpreted the disclosure of Shneiderman . In the Office 
Action mailed December 12, 2005, the Examiner asserted that Figure 4 of Shneiderman teaches 
receiving a user input identifying a group of users to which an archiving user belongs, and the 
user's selection of zero or more persons in the group and zero or more group event types from 
a predetermined plurality of group event types. 

Contrary to the assertions by the Examiner, Shneiderman does not disclose a log in 
process. The application described in Shneiderman permits an operator to enter names as 
freetext and provides a window with previously-stored names, with the most commonly-used 
names listed at the top. See Shneiderman, p. 5, col. 1. Mizoquchi similarly provides for entry 
of names and events by an operator of the device. Neither reference refers to groups of 
users. They certainly do not describe selecting group event types or persons, where the 
selections are based on a group of which the archiving user is a member. 

These arguments were presented to the Examiner during prosecution. See Response to 
Office Action of July 15, 2005; Response to Final Office Action of Feb. 10, 2006. The Examiner 
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has never addressed the arguments, but in reply merely repeated verbatim the previous 
rejections without additional comment. See Office Action of Jan. 26, 2005, p. 3-5; Office Action 
of Dec. 12, 2005, p. 3-4. 

The art cited by the Examiner simply does not teach or suggest the subject matter of 
claim 1. The obviousness rejection of claim 1 is incorrect and should be reversed. 

Dependent Claims 2-13 

Dependent claims 2-13 depend directly or indirectly from claim 1 and recite additional 
limitations which capture other aspects of the invention. Specifically, these claims recite 
elements incorporating the following: 

• retrieving digital media items using a group identified by another user (claim 2); 

• defining distinct groups of people and group event types appropriate for members of 
the groups (claim 3); 

• identifying a digital media item as associated with a "high point" (claims 5 and 6); 

• identifying the media type of a digital media item (claims 7 and 8); 

• associating media items by receiving archiving input data associating multiple digital 
media items, then storing and retrieving media items based on such an association 
(claims 9-12); and 

• performing "automatic nostalgic retrievar of digital media items based on an initial 
selection by the user (claim 13). 

The Examiner seemingly identifies random portions of Shneiderman, and asserts that those 
portions teach the subject matter of the dependent claims. In every case, the Examiner has 
misinterpreted either the subject matter of the claim or the disclosure of Shneiderman . 

With respect to claim 2, the Examiner asserts that Shneiderman teaches the subject 
matter because "user log in with the system which identify the person in the library [sic] (fig 
4)." Office Action of Dec. 12, 2005, p. 5. This rejection simply fails to address elements of the 
claim, specifically that the retrieval input data "comprises a user input from another user 
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identifying a group to which the other user belongs". As previously discussed, there is 
simply no indication that the system described in Shneiderman allows for interaction between 
multiple users in a group. 

With respect to claim 3, the Examiner again cites to Figure 4 and asserts that "each of 
the photo being association with group of users in the photo [sic]/' and thus Shneiderman 
teaches "defining the distinct groups of people, and defining group event types that are 
appropriate for members of the group to distinguish episodic events memorable to the group" 
as recited in claim 3. Office Action of Dec. 12, 2005, p. 5. Clearly, a photo of a group of people 
is "associated with" the people shown in the photo. However, this has no bearing on whether 
the people in the photo are users of the archival system, nor whether the "group" of people 
shown in the photo is in any way related to the "groups" defined in the archival system of the 
present invention. Furthermore, it is unclear how the Examiner believes that photographs 
define group event types as required by the claims. 

Claims 5 and 6 are directed to the use of "high points" in archiving digital media items. 
As described in the present specification at page 4, designating a media item as comprising a 
"high point" allows a user to associate the media item with a "particularly memorable event." 
An example of such an event is a group's last day of school. See application, p. 12. In 
rejecting these claims, the Examiner asserted that archiving media items as high points is 
taught by Shneiderman because, in Figure 4, "time is the high point of the photo." Office 
Action of Dec. 12, 2005, p. 5. The Examiner further stated that retrieving archived media items 
is taught by Shneiderman because a user of the system described in Shneiderman can search 
"by names of people in each photo." Id The Examiner misunderstands the present disclosure, 
the cited reference, or both. Associating a photograph with a specific time as taught in 
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Shneiderman is simply irrelevant to the concept of archiving a digital media item as a 
"particularly memorable event;" searching photographs based on the names of people in the 
photographs is similarly irrelevant to the concept of searching based on "high points/' It is 
unclear what relationship, if any, the Examiner sees between the two types of activities. 

Claims 7 and 8 relate to identifying the type of media item to be archived or retrieved. 
As described in the specification, digital media items can be of any type. Digital media items 
may be sound recordings, video clips, still photographs, etc. In contrast, the system disclosed 
in Shneiderman is used only to annotate digital images. The Examiner asserts that 
Shneiderman teaches identification of media type at col. 3, lines 4-8 and col. 1, lines 4-8. The 
only reference in Shneiderman to media "type" is that professional photographers may annotate 
photographs based on "film type." Shneiderman, p. 1. "Film type" refers to non-digital 
photographs (digital cameras do not use film); any photograph will have the same media type - 
namely, "image." Thus, there is no suggestion in Shneiderman that digital media items can be 
archived or retrieved based on their media type. 

Regarding claims 9-12, the Examiner asserts that these claims are rendered obvious by 
the disclosure of Shneiderman that a photograph may be associated with people and dates, and 
that a set of photographs may be searched by person. See Office Action of Dec. 12, 2005, p. 6. 
However, claim 9, from which claims 10-12 depend directly or indirectly, recites "receiving 
archiving input identifying a plurality of digital media items and an input identifying the digital 
media items to be associated as perceived by the user". Hence, a user may select an 
arbitrary set of media items, and create an association based on the user's perception of those 
items. Shneiderman merely allows a user to annotate a photograph with names of people, 
places, objects, etc. in the photograph. There is no suggestion that a set of photographs may 
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be grouped together based on an arbitrary association. Again, the Examiner has misunderstood 
the claims, and incorrectly applied the cited art. 

Claim 13 allows for nostalgic retrieval of digital media items via an automated process. 
As described in the present specification at pages 4-5, this process allows a user to specify an 
initial starting point, such as a time or event, and then follow a "trail" of media items that is 
determined by the changing selections of persons in the group, event types, and time periods. 
Similarly, the system may provide an initial, randomly selected set of media items, and then 
present further media items based on user input or on further random selections. The 
Examiner asserted that this claim was obvious over the disclosure of Shneiderman that stored 
photographs may be retrieved based on searching names of people in each photograph. See 
Office Action of Dec. 12, 2005, p. 7. It is unclear how this simple search-by-name functionality 
is related to the automated process recited in claim 13. The disclosure of Shneiderman requires 
a user to search stored photographs by a person's name. In contrast, the claimed method 
repeatedly presents a user with digital media items, without requiring initial user input. The 
disclosure of a simple search-by-name function does not render the complex, automated 
nostalgic retrieval process recited in claim 13 obvious. 

As shown above, there is little or no relationship between the disclosure of Shneiderman 
and the elements recited in claims 2-13. As with claim 1, Applicants presented these 
differences during prosecution, but they were not directly addressed by the Examiner. The 
obviousness rejections are incorrect and should be reversed. 
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Claims 14-27 

Independent claims 14, 15, and 17 are apparatus and method claims with limitations 
similar to those in claim 1. Specifically, these claims recite elements incorporating the 
following: 

• "a group to which the user belongs" (claims 14 and 15); 

• "selection of zero or more group event types from a predetermined plurality of group 
event types specific to the group" (claims 14, 15, and 17); and 

• "selection of zero or more persons in the group" (claims 14, 15, and 17). 

As described above with respect to claim 1, neither Shneiderman nor Mizoquchi teaches any 
application of user groups and group event types in archiving digital media items. 

Similarly, dependent claims 16 and 18-27 were rejected by the Examiner on the same 
grounds as their corresponding method claims. As described previously with respect to the 
dependent method claims, Shneiderman and Mizoquchi do not teach or suggest the subject 
matter of these claims. In many instances, the references are not relevant to the subject 
matter. As with the previous claims, in response to Applicants' arguments the Examiner merely 
repeated the previous rejection without additional comment. See Office Action of Jan. 26, 
2005, p. 3-7; Office Action of Dec. 12, 2005, p. 5-7. The rejection of claims 14-27 is incorrect, 
and should be reversed. 

THE EXAMINER APPLIED AN IMPROPER STANDARD IN COMPARING THE CITED ART 
TO THE PRESENT INVENTION AND IGNORED MOTIVATION IN THE ART 

In an interview conducted February 9, 2006, the Examiner maintained that the failure of 
the cited art to describe the use of groups is a minor omission. While Applicants respectfully 
disagree with the Examiner's characterization of the art, this point of view ignores the legal 
standard for obviousness. Under the proper standard, the Examiner was required to find prior 
art that disclosed all the elements of the claims, not just the "important" elements. M.P.E.P. § 
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2142. In addition the specification clearly explains that the use of group-based indexing and 
retrieval operations can provide significant benefits. See pp. 1-3 and throughout the 
specification. The claims fully meet the legal requirements for patentability. 

In addition, the Examiner has not provided, nor is there, a motivation to combine 
Shneiderman, Mizoquchi, and/or Astle . To establish a prima facie case of obviousness, there 
must be some suggestion or motivation, either in the references themselves or in the 
knowledge generally available to one of ordinary skill in the art, to modify the reference or to 
combine reference teachings. The teaching or suggestion to make the claimed combination and 
the reasonable expectation of success must both be found in the prior art, and not based on 
applicant's disclosure. Vaeck, 947 F.2d 488; MPEP § 2142-43. There is no motivation to 
combine the annotation system of Shneiderman with the video sequence indexing system of 
Astle . Astle discloses the use of time counters or index frames to index video sequences. Astle, 
col. 2, lines 11-25; col. 6, lines 3-50. A person of ordinary skill in the art would have no reason 
to adapt either indexing system to the annotation system of Shneiderman . Shneiderman 
already provides a means of recording the date of a photograph. Shneiderman, p. 4. The 
frame-based indexing system of Astle would have no meaning in indexing photographs as 
disclosed by Shneiderman , since each photograph is already a separate "frame." 

Similarly, there is no motivation to combine the in-camera annotation system of 
Mizoquchi with the computer-based annotation system of Shneiderman . Neither reference 
indicates that it is desirable or possible to combine the disclosed annotation system with the 
system of the other. In fact, one of skill in the art would be unlikely to combine the disparate 
systems, since each uses different techniques, labels, and user interfaces for annotating figures. 
The system of Mizoquchi requires a scheduling/calendaring system to organize photographs, 
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while the system of Shneiderman allows for freetext input. See Mizoquchi, col. 5, lines 1-7; 
col. 8, lines 15-47; Shneiderman, p. 4. One of skill in the art would have no reason to combine 
the schedule-based system of Mizoquchi with the freetext annotation system of Shneiderman . 

In each case, the Examiner has failed to point out where in the references or in the 
knowledge generally available to one of skill in the art there is a motivation to combine any of 
the cited references. The obviousness rejections of all the claims are incorrect, and must be 
reversed. 
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CONCLUSION 

Applicant respectfully requests reversal of the obviousness rejection to claims 1-27 and 
58-67. These claims are allowable over the cited art. 

The Commissioner is authorized to charge the fee for this appeal brief of $250.00 (small 
entity) as set forth in 37 C.F.R. §41. 20(b)(2) and all other fees associated with this submission 
to Deposit Account No. 11-0600. 



KENYON & KENYON LLP 
1500 K Street, N.W. 
Washington, D.C. 20005 
Ph.: (202) 220-4200 
Fax.: (202) 220-4201 



Respectfully submitted, 



Date: Mav 15. 2006 




Registration No. 39,702 
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CLAIMS APPENDIX 

1. A method of archiving and retrieving digital media items, comprising 

receiving a user input identifying a group of users to which an archiving user belongs; 

receiving archiving input data identifying: a digital media item to be archived for the 
group, the user's selection of zero or more group event types from a predetermined plurality of 
group event types specific to the group, the user's selection of zero or more persons in the 
group, and the user's selection of a time period; 

generating index information using the received user archiving input; 

storing the index information in association with the identified digital media item; 

repeating the reception of archiving input data, the generation of the index information 
and the storing of the index information for a plurality of digital media items; 

receiving retrieval input data representing a selection of a default or zero or more group 
event types from the predetermined plurality of group event types for the group, a selection of 
a default or zero or more persons in the group, and a selection of a time period; and 

using the selections and the identified group to retrieve and output digital media items 
that match the selection. 

2. A method according to Claim 1 wherein the retrieval input data comprises a user input from 
another user identifying a group to which the other user belongs and the digital media items 
are retrieved using the group identified for the other user in the user retrieval input. 

3. A method according to claim 1 including defining the distinct groups of people, and defining 
group event types that are appropriate for members of the groups to distinguish episodic 
events memorable to the group. 
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4. A method according to claim 1 including receiving said digital media item to be archived, and 
storing said digital media item in association with the index information. 

5. A method according to claim 1 including receiving archiving input data identifying a digital 
media item as being associated with a memorable high point in the mind of the user. 

6. A method according to claim 5 wherein the retrieval input data includes an input selecting 
memorable high points. 

7. A method according to claim 1 wherein the index information includes an identification of a 
media type of the digital media item. 

8. A method according to claim 7 wherein the retrieval input data includes an input identifying 
a media type, and the digital media items are retrieved and output based on the identified 
media type. 

9. A method according to claim 1 including receiving archiving input data identifying a plurality 
of digital media items and an input identifying the digital media items to be associated as 
perceived by the user, wherein the index information is generated to include the identified 
association. 

10. A method according to claim 9 wherein, when digital media items are retrieved and output 
as a result of the user retrieval input, any digital media items having the identified association in 
the index information are automatically identified for retrieval and output. 

11. A method according to claim 10 wherein the automatically identified digital media items are 
automatically retrieved and output. 
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12. A method according to claim 10 including outputting a notification to a user that associated 
digital media items are available, and retrieving and outputting automatically identified digital 
media items in response to a user input. 

13. A method according to claim 1 further comprising: 

receiving a user request for automatic nostalgic retrieval, 

automatically generating an initial set of said selections, 

using the selections to retrieve and output digital media items, 

automatically modifying one or more of the selections, 

using the modified selections to retrieve and output digital media items and 

repeating the modifying, and retrieval and output steps. 

14. A user terminal for use in the archiving and retrieval of digital media items associated with 
predefined distinct groups of one or more people, the terminal comprising: 

user interface means for generating archiving input data identifying: 
a group to which the user belongs, 
a digital media item to be archived for the group, 

a user selection of zero or more group event types from a predetermined 
plurality of group event types specific to the group, 

a user selection of zero or more persons in the group, and 
a user selection of a time period; 
transmission means for transmitting the archiving input to a processing device for 
generating index information using the archiving input and for storing the index information in 
association with the identified item; 
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wherein said user interface means further is for generating retrieval input data 
identifying: 

a group to which a retrieving user belongs, 

a retrieving user's selection of a default or zero or more group event types from 
the predetermined plurality of group event types for the group, 

a retrieving user's selection of a default or zero or more persons in the group, 

a retrieving user's selection of a time period; and 
said transmission means is adapted to transmit the retrieval input to the processing 
device to identify digital media items using the retrieval input; 
the user terminal further including 

receiving means for receiving any digital media items identified by the processing 
device; and 

a display for displaying the received digital media items. 

15. A method of operating a terminal for use in the archiving and retrieval of digital media 
items for predefined distinct groups of people, the method comprising: 
receiving from an archiving user archiving input data identifying: 
a group to which the user belongs, 
a digital media item to be archived for the group, 

a selection of zero or more group event types from a predetermined plurality of 
group event types specific to the group, 

a selection of zero or more persons in the group, and 
a selection of a time period; 
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transmitting the archiving input to a processing device for generating index information 
using the archiving input and for storing the index information in association with the identified 
item; 

receiving from a retrieving user retrieval input data identifying: 
a group to which the user belongs, 

a selection of default or zero or more group event types from the predetermined 
plurality of group event types for the group, 

a selection of default or zero or more persons in the group, and 

a selection of a time period; 
transmitting the retrieval input to the processing device to identify digital media items 
using the retrieval input; 

receiving any digital media items identified by the processing device; and 
displaying the received digital media items. 

16. A carrier medium storing processor readable and implementable code for controlling a 
processor to carry out the method of any one of claims 1 to 13 or 15. 

17. Apparatus for archiving and retrieving digital media items for predefined distinct groups of 
one or more people, the apparatus comprising: 

receiving means for receiving archiving input data identifying a group to which the user 
belongs, the archiving input data identifying: 

a digital media item to be archived for the group, 

a selection of zero or more group event types from a predetermined plurality of 
group event types specific to the group, 

a selection of zero or more persons in the group, and 
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a selection of a time period; 
generating means for generating index information using the received user archiving 

input; 

storing means for storing the index information in association with the identified digital 
media item; 

wherein said receiving means is adapted to receive retrieval input data identifying a 
manual or automatic selection of zero or more group event types from the predetermined 
plurality of group event types for the group, a selection of zero or more persons in the group, 
and a selection of a time or time period; and 

the apparatus further includes retrieval means for using the selections and the identified 
group to retrieve and output digital media items that match the selections. 

18. Apparatus according to claim 17 wherein said receiving means is adapted to receive the 
retrieval input data from another user, said retrieval input data identifying a group to which the 
other user belongs. 

19. Apparatus according to claim 17 including means for defining the distinct groups of people, 
and for defining group event types that are appropriate for members of the groups to 
distinguish episodic events memorable to the group. 

20. Apparatus according to claim 17 wherein said receiving means is adapted to receive said 
digital media items to be archived, and item storing means for storing said digital media item in 
association with the index information. 
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21. Apparatus according to claim 17 wherein said receiving means is adapted to receive 
archiving input data identifying a digital media item as being associated with a memorable high 
point in the mind of the user. 

22. Apparatus according to claim 21 wherein said receiving means is adapted to receive 
retrieval input data selecting memorable high points. 

23. Apparatus according to claim 17 wherein said generating means is adapted to include an 
identification of a media type of the digital media item. 

24. Apparatus according to claim 23 wherein said receiving means is adapted to receive 
retrieval input data identifying a media type, and said retrieval means is adapted to retrieve and 
output digital media items based on the identified media type. 

25. Apparatus according to claim 17 wherein said receiving means is adapted to receive 
archiving input data identifying a plurality of digital media items to be sequenced as perceived 
by the user, and said generating means is adapted to generate the index information to include 
the identified sequences. 

26. Apparatus according to claim 25 wherein said retrieval means is adapted to retrieve all 
digital media items identified to be sequenced when one or more digital media items are 
selected for retrieval. 

27. Apparatus according to claim 17 wherein said receiving means receives a request for 
automatic nostalgic retrieval, said generating means is adapted to generate an initial set of 
selections and automatically modify one or more of the selections at a time in response to the 
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request, said retrieval means is adapted to sequentially output digital media items retrieved 
using the generated and modified sets of selection. 

28-57. (cancelled) 

58. A media archival method, comprising, under control of an operator who is a member of a 
group: 

authenticating an operator as a member of a group of users, 

identifying candidate identification values based upon the group with whom the operator 
is authenticated, 

querying the operator for selection of identification data to be associated with a digital 
media item, the query identifying the candidate identification values and including valid 
selections of an event type and persons from the group and time, 

generating index information from a response of the operator, and 
storing the index information in association with the digital media item. 

59. The archival method of claim 58, wherein the candidate identification values for persons 
include names of group members. 

60. The archival method of claim 58, wherein the stored index information includes a flag that 
distinguishes high point items from other items, and the method further comprises setting the 
flag if the operator response includes an indication that the digital media item is a high point. 

61. The archival method of claim 58, wherein, if the operator response indicates that the digital 
media item is a member of a trail, the index information includes an identifier representing the 
media item's display position in a sequence of stored media items stored by the system. 
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62. A multimedia retrieval method, comprising, under control of an operator who is a member 
of a group: 

authenticating an operator as a member of a group of users, 

identifying candidate identification values based upon the group with whom the operator 
is authenticated, 

querying the operator for selection of identification data, the query identifying the 
candidate identification values and including valid selections of an event type and persons from 
the group and time, 

generating index information from a response of the operator, and 
retrieving stored media items corresponding to the index information. 

63. The retrieval method of claim 62, wherein the candidate identification values for persons 
include names of group members. 

64. The retrieval method of claim 62, wherein the stored index information includes a flag that 
distinguishes high point items from other items, and the method further comprises searching for 
the flag among the stored index information if the operator response includes an indication that 
high point items are selected. 

65. The retrieval method of claim 62, further comprising, if the operator response indicates 
that a trail is selected, presenting stored media items in a sequence as identified in the index 
information corresponding to the presented media items. 

66. A method of archiving digital media items, comprising: 

receiving a user input identifying a social group to which an archiving user belongs; 
building a database that includes: 
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digital media items to be archived for the social group, and 
index information for the digital media items, each instance of index information 
created from archiving input data identifying a user's response to a query that identifies 
a plurality of event types previously registered as associated with the social group, and 
persons previously registered as members of the social group. 

67. A method of searching digital media items, comprising: 

receiving a user input identifying a social group for which a search is to be conducted; 

identifying candidate identification values based upon the social group, 

displaying a query that identifies the candidate identification values and including valid 
selections of an event type for the social group, persons from the social group and time, 

responsive to selection criteria made in response to the query, searching a database and 
retrieving digital media items that satisfy the selection criteria. 
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EVIDENCE APPENDIX 

The following items were made of record and have been relied on by the Examiner in 
making the appealed rejections: 

Shneiderman, et al., "Direct Annotation: A Drag-and-Drop Strategy for Labeling Photos/' 
Proc. International Conference Information Visualizaion (IV2000), London, England. 

Mizoguchi, Yoshiyuki, "Information Processing Method and Apparatus Therefore/' E.P. 
Application No. 0 678 816 A2, Oct. 25, 1995. 

Astle, Brian, "Video Database Indexing and Method of Presenting Video Database Index 
to a User/' U.S. Patent No. 5,485,611, Jan. 16, 1996. 
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RELATED PROCEEDINGS APPENDIX 



None. 
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Direct Annotation: 
A Drag-and-Drop Strategy for Labeling Photos 



Ben Shneiderman, Hyunmo Kang 
Dept. of Computer Science, Human-Computer Interaction Laboratory, 
Institute for Advanced Computer Studies & Institute for Systems Research 
University of Maryland, College Park, MD 20742 USA 
{ben, kan£}(5),cs. umd.edu 



Abstract 

Annotating photos is such a time-consuming, tedious 
and error-prone data entry task that it discourages most 
owners of personal photo libraries. By allowing users to 
drag labels such as personal names from a scrolling list 
and drop them on a photo, we believe we can make the 
task faster, easier and more appealing. Since the names 
are entered in a database, searching for all photos of a 
friend or family member is dramatically simplified. We 
describe the user interface design and the database 
schema to support direct annotation, as implemented in 
our PhotoFinder prototype. 

Keywords: direct annotation, direct manipulation, 
graphical user interfaces, photo libraries, drag-and-drop, 
label placement 

1. Introduction 

Adding captions to photos is a time-consuming and 
error prone task for professional photographers, editors, 
librarians, curators, scholars, and amateur photographers. 
In many professional applications, photos are worthless 
unless they are accurately described by date, time, 
location, photographer, title, recognizable people, etc. 
Additional annotation may include details about the photo 
(for example, film type, print size, aperture, shutter speed, 
owner, copyright information) and its contents (keywords 
from controlled vocabularies, topics from a hierarchy, free 
text descriptions, etc.)- For amateur photographers, 
annotations are rarely done, except for the occasional 
handwritten note on the back of a photo or an envelope 
containing a collection of photos. 

For those who are serious about adding annotations, 
the common computer-based approach is to use database 
programs, such as Microsoft Access, that offer form fill-in 
or free text boxes and then store the information in a 
database. Data entry is typically done by typing, but 
selecting attribute values for some fields (for example, 
black&white or color film) is supported in many systems. 



Of course, simpler tools that provide free-form input, such 
as word processors, spreadsheets, and other tools are used 
in many situations. Captions and annotations are often 
displayed near a photo on screen displays, web pages, and 
printed versions. Software packages (Kodak PhotoEasy, 
MGI PhotoSuite, Aladdin Image AXS, etc.) and web sites 
(Kodak's photonet, Gatherround.com, shutterfly, etc.) 
offer modest facilities to typing in annotations and 
searching descriptions. 

As photo library sizes increase the need and benefit of 
annotation and search capabilities grows. The need to 
rapidly locate photos of Bill Clinton meeting with Boris 
Yeltsin at a European summit held in 1998 is strong 
enough to justify substantial efforts in many news 
agencies. More difficult searches such as "agriculture in 
developing nations" are harder to satisfy, but many web 
and database search tools support such searches (Lycos, 
Corbis, etc.). Query-By-Image-Content from IBM, is one 
of many projects that uses automated techniques to 
analyze image ( http ://wwwqbic . almaden . ibm. com/ ) . 
Computer vision techniques can be helpful in finding 
photos by color (sunsets are a typical example), 
identifying features (corporate logos or the Washington 
Monument), or textures (such as clouds or trees), but a 
blend of automated and manual techniques may be 
preferable. Face recognition research offers hope for 
automated annotation, but commercial progress is slow 
[IP]- 

2. Related Work on Annotation 

Annotation of photos is a variation on previously 
explored problems such as annotation on maps [3] [4] [5] 
in which the challenge is to place city, state, river, or lake 
labels close to the features. There is a long history of 
work on this problem, but new possibilities emerge 
because of the dynamics of the computer screen (Figure 
1). However, annotation is usually seen as an authoring 
process conducted by specialists and users only chose 



whether to show or hide annotations. Variations on 
annotation also come from the placement of labels on 
markers in information visualization tasks such as in tree 
structures, such in the hyperbolic tree [6] (Figure 2) or in 
medical histories, such as LifeLines [7] (Figure 3). 
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Figure 2. Hyperbolic Tree 
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Figure 3. LifeLines Medical Patient History 

Previous work on annotation focused on writing 
programs to make label placements that reduced overlaps 
[8], but there are many situations in which it is helpful for 
users to place labels manually, much like post-it notes, on 
documents, photos, maps, diagrams, webpages, etc. 
Annotation of paper and electronic documents by hand is 
also a much-studied topic with continuing innovations [9]. 
While many systems allow notes to be placed on a 
document or object, the demands of annotating personal 
photo libraries are worthy of special study [10]. We 
believe that personal photo libraries are a special case 
because users are concentrating on the photos (and may 
have a low interested in the underlying technology), are 
concerned about the social aspects of sharing photos, and 
are intermittent users. They seek enjoyment and have 
little patience for form filling or data entry. 

3. The PhotoFinder Project 

In the initial stages of our project on storage and 
retrieval from personal photo libraries 
( http://www.cs.umd.edu/hcil/photolib/ ), we emphasize 
collection management and annotation to support 
searching for people. This decision was based on our user 
needs assessment, reports from other researchers, and our 
personal experience that indicate that people often want to 
find photos of a friend or relative at some event that 
occurred recently or years ago [2][11]. Personal photo 
libraries may have from hundreds to tens of thousands of 
photos, and organization is, to be generous, haphazard. 
Photos are sometimes in neat albums, but more often put 
in a drawer or a shoebox. While recent photos are often 
on top, shuffling through the photos often leaves them 
disorganized. Some users will keep photos in the 
envelopes they got from the photo store, and more 
organized types will label and order them. 



As digital cameras become widespread, users have had 
to improvise organization strategies using hierarchical 
directory structures, and typing in descriptive file and 
directory names to replace the automatically generated 
photo file numbers. Some software packages (PhotoSuite, 
PhotoEasy, etc.) enable users to organize photos into 
albums and create web pages with photos, but annotation 
is often impossible or made difficult. Web sites such as 
Kodak's PhotoNet.com, Gatherround.com, etc. enable 
users to store collections of photos and have discussion 
groups about the collections, but annotation is limited to 
typing into a caption field. The pioneering effort of the 
FotoFile [2] offered an excellent prototype that inspired 
our work. 

Our goal in the PhotoFinder project was to support 
personal photo library users. We developed a conceptual 
model of a library having a set of collections, with each 
collection having a set of photos. Photos can participate 
in multiple collections. Collections and individual photos 
can be annotated with free text fields plus date and 
location fields stored in a database (see Figure 6 for our 
Photo Library database schema). Our interface has three 
main windows: 

■ Library viewer: Shows a representative photo for 
each collection, with a stack representing the number 
of photos in each collection. 

■ Collection viewer: Shows thumbnails of all photos in 
the collection. Users can move the photos around, 
enlarge them all or individually, cluster them, or 
present them in a compact manner. A variety of 



thumbnail designs were prototyped and will be 
refined for inclusion in future versions. 
■ Photo viewer: Shows an individual photo in a 
resizable window. A group of photos can be selected 
in the Collection viewer and dragged to the Photo 
viewer to produce an animated slide show. 
We also put a strong emphasis on recording and 
searching by the names of people in each photo. We 
believed that a personal photo library might contain 
repeated images of the same people at different events, 
and estimated 100-200 identifiable people in 10,000 
photos. Furthermore we expected a highly skewed 
distribution with immediate family members and close 
friends appearing very frequently. The many-to-many 
relationship between photos and people is mediated by the 
Appearance relation (Figure 6) that stores the 
identification of all the people who appear in each photo. 

Such a database would support accurate storage of 
information, but we recognized that the tedious data entry 
problem would prevent most users from typing in names 
for each photo. Furthermore, the inconsistency in names 
is quickly a problem with misspellings or variant names 
(for example, Bill, Billy, William) undermining the 
success of search. 

A second challenge we faced was that the list of names 
of people appearing in a photo could often be difficult to 
associate with individuals, especially in group shots. 
Textual captions often indicate left-to-right ordering in 
front and back rows, or give even more specific 
identification of who is where. 




Figure 4. PhotoFindeM display with Library Viewer on the left, Collection Viewer with thumbnails on 
the upper right, and Photo Viewer on the lower right. 



4. Direct Annotation 

To cope with these challenges we developed the 
concept of direct annotation: selectable, dragable labels 
that can be placed directly on the photo. Users can select 
from a scrolling or pop-up list and drag by mouse or touch 
screen. This applies direct manipulation principles [12] 
that avoid the use of a keyboard, except to enter a name 
the first time it appears. The name labels can be moved or 
hidden, and their presence is recorded in the database in 



the Appearance relation with an X-Y location, based on 
an origin in the upper left hand corner of the photo. 
This simple rapid process also allows users to annotate at 
will. They can add annotations when they first see their 
photos on the screen, when they review them and make 
selections, or when they are showing them to others. This 
easy design and continuous annotation facility may 
encourage users to do more annotation. Figures 5 (a)-(f) 
show the process of annotation on a set of four people at a 
conference. 
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(e) Four Identified People 

Figure 5. The Process of Dragging and 

The selection list is shown as being an alphabetically 
organized scrolling menu, but it could be implemented as 
a split menu [13]. This would entail having 3-5 of the 
most commonly occurring names in a box, followed by 
the alphabetical presentation of the full list. Thus the 
most frequent names would be always visible to allow 
rapid selection. Name completion strategies for rapid 
table navigation would be useful in this application. 
When users mouse down on a name, the dragging begins 
and a colored box surrounds the name. When users 
mouse up, the name label is fixed in place, a tone is 
sounded, and the database entry of the XY coordinates is 



(f) Hide Annotations 

Dropping an Annotation on a Photo 

stored. The tone gives further feedback and reinforces the 
sense of accomplishment. Further reinforcement for 
annotation is given by subtly changing the border of 
photos in the Collection viewer. When a photo gets an 
annotation, its thumbnail's white border changes to green. 
Users will then be able to see how much they have 
accomplished and which photos are still in need of 
annotation. 

A Show/Hide checkbox gives users control over seeing 
the photo with and without the name labels. Since the 
photo viewer window is resizable, the position of the 
labels changes to make sure they remain over the same 



person. A small marker (ten pixels long) hangs down 
from the center of the label to allow precise placement 
when there are many people close together. The marker 
can be used to point at the head or body and it becomes 
especially useful in crowded group photos. 

Future additions might include the capacity to resize 
the labels, change fonts, change colors, or add animations. 
Another interesting issue is collaborative annotation in 
which multiple users working side-by-side [14] or 
independently might annotate photos and then the results 
could be combined, with appropriate resolution of 
conflicts. Tools for finding variant spellings or switches 
between last and first names would help raise data quality. 
A valuable accelerator is bulk annotation [2], in which a 
group of photos is selected and then the same label is 
applied to every photo with one action, although 
individual placement might still be needed. 

Of course, annotation by names of people in photos is 
only the first step. Drag and drop annotation for any kind 
of object in a photo (car, house, bicycle), map (cities, 
states, lakes), or painting (brushstroke, signature, feature) 
is possible. Annotation about the overall image, such as 
type of photo (portrait, group, landscape), map (highway, 
topographic, urban), or painting (impressionist, abstract, 
portrait) is possible. Colored ribbons or multiple star 
icons could be used to indicate the importance or quality 
of photos. 

Searching and browsing become more effective once 
annotations are included in the photo database. The 
obvious task is to see all photos that include an individual. 
This has been implemented by simply dragging the name 
from the list into the Collection viewer or to a designated 
label area. The PhotoFinder finds and displays all photos 
in which that name appears in a label. 

5. Database Design and Implementation 
5.1 Schema of the Photo Library database 



The PhotoFinder operates using a Photo Library 
database (Microsoft Access), which contains five linked 
tables (Figure 6). The basic concept is that a Photo 
Library contains Collections of Photos, and that Photos 
contain images of People. 

In the Photo Library schema, the Collections table 
represents the collections of photos with attributes such as 
Collection Title, Description, Keywords, Starting Date, 
Ending Date, Location, Representative PhotoID and 
unique Collection ID. The Photos table is where 
references (full path and file name) of photos and their 
thumbnails are stored with important attributes such as the 
date of photo, event, keywords, location, rating, color, 
locale, and so on. Each photo should have a unique 
reference and photos with the same references are not 
allowed to be stored in this table even though they have 
different attribute values. The Linkage table is the 
connection between the Collections table and Photos 
table. It stores the links between collections and photos. 

The People table stores all the information about the 
people who appear in the Photo Library. In our initial 
implementation, attributes include only the Given (First) 
name and Family (Last) name of the person, and the 
unique PersonID (people with the same first and last name 
are not allowed to be stored in People table). Eventually, 
the People table will be extended to include personal 
information such as e-mail address for exporting the Photo 
Library, homepage address, occupation and so on. The 
Appearance table stores the information about which 
Person is in which Photo. It serves as the linkage between 
the Photos table and the People table. Attributes include 
AppearancelD, PersonID, PhotoID, and relative (X, Y) 
coordinates (upper left corner is (0,0), lower right is 
(100,100)) of people in the photos. 

In designing the Photo Library, we made three major 
assumptions concerning the Library, Collections, and 
Photos. These assumptions can be classified as follows: 



Figure 6. The schema of Photo Library database 



■ Relationship between Collections and Photos 

A 1 -to-many relationship between the Collections table 
and the Linkage table has been set so that a collection can 
contain multiple photos, and a 1 -to-many relationship 
between the Photos table and the Linkage table has been 
set so that same photo can be included in multiple 
collections. It is also possible that a collection contains the 
same photo multiple times to permit reappearances in a 
slide presentation. Two different collections could have 
exactly same set of photos. If two photos have different 
path names, they are different photos even though they are 
copies of a photo. 

■ Relationship between Photos and People 

A 1 -to-many relationship between the Photos table and 
the Appearance table has been set so that a photo can 
contain multiple persons, and a 1 -to-many relationship 
between People table and Appearance table has been set 
so that same person can be included in multiple photos. 
Multiple appearances of the same person in a photo are 
not allowed. A composite pair of Given name and Family 
name should be unique in the People table. 

■ Relationship among Library, Collections, and Photos 
Within a library, the same photo could be contained in 

multiple collections multiple times, but their attributes and 
annotations must be the same. 



In the first design of the Photo Library database, we 
only considered annotation by names of people in photos, 
but the Photo Library database can be easily extended by 
adding an Object table, Animal table, Keyword table, and 
so on, along with connection tables similar to the 
Appearance table. With such a Photo Library database 
design, more flexible annotation would be possible. 
5.2 Updating the Photo Library Database by 
Direct Annotation 

PhotoFinder keeps updating the Photo Library database 
whenever the direct annotation module causes any 
information changes. In this section, we classify the Photo 
Library database updating situations into five categories, 
and discuss corresponding algorithm and implementation 
issues. 



■ Adding a New Name Label / Creating a New Person: 
When users drag a name from "People in Library" 
listbox and drop it onto a photo, PhotoFinder immediately 
checks whether there already exists an Appearance 
connection between the photo and the person since 
multiple appearances of the same person in a photo are 
not allowed. If a conflict occurs, PhotoFinder would 
highlight the existing name label on the photo and ignore 
the drag-and-drop event with a warning message. If there 
is no conflict, PhotoFinder finds the PersonID and 
PhotoID, calculates a relative (X, Y) position (0< X, Y 



<100) of the drag-and-drop point on the photo, and then 
creates a new Appearance record with this information. 
After adding a new record to the Appearance table, the 
PhotoFinder updates "People in this Photo" listbox and 
finally creates a name label on the photo. To show that the 
label has just been inserted, the newly added name in the 
"People in this Photo" listbox will be selected, and 
accordingly the new name label on the photo will be 
highlighted. If the added name label is the first one on the 
photo, PhotoFinder sends an event to the Collection 
Viewer to change the border color of the corresponding 
thumbnail to green, in order to show that the photo now 
has an annotation. The algorithm for creating a new 
person is simple. As soon as users type in the first name 
and last name of a person in the editbox and press enter, 
PhotoFinder checks whether the name already exists in the 
People table. If so, a warning message will be displayed 
with the name in "People in Library" listbox being 
selected. If not, PhotoFinder creates and adds a new 
Person record to the People table, and then updates the 
"People in Library" listbox, selecting and highlighting the 
newly added name. 

■ Deleting Name Label /Deleting Person: 

When the delete button of the Photo Viewer toolbar is 
clicked or the delete key is pressed, PhotoFinder checks 
whether the selected name label already exists. If not, 
PhotoFinder ignores the deleting action. But if it exists, 
PhotoFinder automatically calculates the PersonID of the 
selected name label and the PhotoID, and it searches 
through the Appearance table to find and delete an 
Appearance record having those IDs. PhotoFinder updates 
"People in this Photo" listbox and deletes the name label 
on the photo. If the deleted name label was the last one on 
the photo, PhotoFinder sends an event to the Collection 
Viewer to change the border color of the corresponding 
thumbnail to white, to show that the photo has no 
annotation. If focus is on the "People in Library" listbox 
and the delete key is pressed, PhotoFinder finds the 
PersonID of the selected name in the listbox. PhotoFinder 
deletes the PersonID from the People table and also 
deletes all the Appearance records containing that 
PersonID, which results in the complete elimination of the 
name label from the other photos in the Photo Library. 
Again, Collection Viewer updates the border color of 
thumbnails that no longer have annotations. 

■ Editing a Name of Person: 

Users can edit a name of person in library by pressing 
the edit button of the Photo Viewer toolbar or by just 
double clicking over the selected name in the "People in 
Library" listbox. When the edited name is typed in, 
PhotoFinder finds and changes the corresponding person 



record from the People table only if there is no duplication 
of the name in the People table. It also refreshes both the 
"People in this Photo" and the "People in Library" 
listboxes, and all the name labels on the current photo. If 
duplication occurs, the whole editing process will be 
ignored with a warning message. 

■ Positioning Name Label: 

Drag-and-dropping the existing label over the photo 
can change position of the name label. As mentioned 
before, the relative (X, Y) position of the center point of a 
name label is stored in the corresponding Appearance 
record. PhotoFinder uses a small marker hanging down 
from the center of the label to allow precise placement. 
But since the size and direction (downward) of the marker 
is fixed, it is somewhat difficult to distinguish labels when 
many people appear in the photo close together. Using 
Excentric labels [15] or adding an additional (X, Y) field 
to the Appearance table to allow a longer and directional 
marker could solve this problem. Other features such as 
changing the font size of labels and avoiding occlusion 
among labels in resizing the photo will be handled in 
future versions of PhotoFinder. 

■ Importing People Table from other Libraries: 
Retyping the names that already exist in other libraries 

is very tedious and time-consuming job. Therefore, 
PhotoFinder supports a function to import the People 
table from other libraries. The internal process of 
importing the People table is similar to that of creating a 
new person repeatedly. The only thing PhotoFinder should 
handle is checking and eliminating the duplication of a 
person name. 

6. Conclusion 

Digital photography is growing rapidly, and with it the 
need to organize, manage, annotate, browse and search 
growing libraries of photos. While numerous tools offer 
collection or album management, we believe that the 
addition of easy to use and enjoyable annotation 
techniques is an important contribution. After a single 
demonstration, most users understand direct annotation 
and are eager to use it. We are adding features, 
integrating search functions, and conducting usability 
tests. 
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time range in which its photographic time is 
included is added to the image data (S5) and 
stored. An image data search operation is per- 
formed on the basis of the stored event infor- 
mation. 
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The present invention relates to an information 
processing method of recording and reproducing 
electronic information and an apparatus therefor. 

In conventional photography using a silver salt 
film, along with developments of camera techniques 
including automatic focus control (AF) and automatic 
exposure control (AE), a photograph intended by a 
photographer has been able to be taken with an easy 
operation. To search a desired photograph from the 
photographs taken by the photographer, it is impor- 
tant that the photographs taken by the photographer 
are properly filed so as to determine easy access to 
the desired photograph. This filing entirely depends 
on the efforts of the photographer. 

More specifically, even if there are a large num- 
ber of excellent photographs which reflect the will of 
the photographer, improper filing may hoard these 
photographs. No drastic countermeasures for filing 
photographs have been made in the conventional 
photographic system using silver salt films, except for 
a simple countermeasure such as date imprinting. 
For this reason, the photographer must file his pho- 
tographs in an album or the like and record the con- 
tents of the photographs, resulting in cumbersome 
operations. 

No system improvement has been made to solve 
the above problem even in an electronic still camera 
for recording a photograph as an electrical signal on 
a floppy disk. Information added to image information 
includes the date, the serial number, and the like. For 
this reason, the photographic contents must be de- 
scribed in the recorded floppy disk and managed. 
These data added to the image information are super- 
posed on analog signals serving as image data. To re- 
write these data or add new data, the image data itself 
must also be rewritten to require a large circuit size. 
In addition, it is impossible to access these data in a 
search mode. It is, therefore, difficult to perform a 
high-speed search operation. 

In recent years, an electronic still camera for re- 
cording a photograph as a digital signal in a memory 
card is proposed and put into practical use. In this still 
camera, data for searching a photographed image 
can be added easier than in the conventional system. 
An embodiment disclosed in Japanese Patent La id- 
Open No. 4-70735 proposes a method of managing 
and searching image data 

In this method, any one of a photographic date, a 
photographic place, weather, a name of person, a 
portrait, an event, and audio data is recorded as im- 
age search data together with image data. The image 
search data can be input together with the image data 
automatically or by the photographer upon comple- 
tion of photography. Subsequently, a data type 
(search key) is designated. When this data is input, an 
image having data whose contents are identical to 
those of the input data is searched. For example, if a 
person is designated and "Yoshlda" is input, an image 



having portrait data representing a person "Yoshida" 
is searched and output If time is designated, and a 
specific moment is input, an image having the specif- 
ic moment as the data is searched and output. 
5 In consideration of personal use, the most effec- 

tive search key of the above search keys in the search 
mode is the event key representing that a photograph 
is taken in a 'Yield day", a "golf course", or the like. No 
general method of inputting event data is described in 
10 the above prior art As a special case, there is descri- 
bed a method of transmitting the above data in only 
a limited case by means of an electric wave and caus- 
ing a camera to receive and automatically record the 
electric wave as image data. In this case, the applt- 
15 cation range of the camera is undesirably limited. 

The present invention has been made in view of 
the conventional problems described above, and has 
as one object to provide an information equipment, 
capable of filing photographed images during photog- 
20 raphy, recording management data, and efficiently 
searching a photographed image. 

According to the present invention, there is pro- 
vided an information processing method comprising: 
an input step of inputting first data; and 
25 a step of storing predetermined third data and 

the first data input in the input step so as to associate 
the predetermined third data with the first data when 
second data associated with the first data input in the 
input step is identical to the predetermined third data, 
30 wherein the first data is searched on the basis 

of the stored third data. 

In another aspect, the invention provides an infor- 
mation processing method comprising: 

an input step of inputting first data; and 
35 a step of storing predetermined third data and 

the first data input in the input step so as to associate 
the predetermined third data with the first data when 
second data associated with the first data input in the 
input step is included in a category of the predeter- 
40 mined third data, 

wherein the first data is searched on the basis 
of the stored third data. 

In yet another aspect the invention provides an in- 
formation processing method comprising: 
45 an input step of inputting first data; and 

a step of storing predetermined fourth data as- 
sociated with predetermined third data and the first 
data input in the input step so as to associate the pre- 
determined fourth data with the first data when sec- 
50 ond data associated with the first data input in the in- 
put step is identical to the predetermined third data, 
wherein the first data is searched on the basis 
of the stored fourth data. 

In yet another aspect the invention provides an in- 
55 formation processing method comprising: 

an input step of inputting first data; and 
a step of storing predetermined fourth data as- 
sociated with predetermined third data and the first 
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data input in the input step so as to associate the pre- 
determined fourth data with the first data when sec- 
ond data associated with the first data input in the in- 
put step is included in a category of the predeter- 
mined third data, 5 

wherein the first data is searched on the basis 
of the stored fourth data. 

According to one embodiment, the foregoing ob- 
ject is attained by providing an information processing 
apparatus comprising: 10 
input means for inputting first data; and 
means for storing predetermined third data and 
the first data input by said input means so as to as- 
sociate the predetermined third data with the first 
data when second data associated with the first data 15 
input by said input means is identical to the predeter- 
mined third data, 

wherein the first data is searched on the basis 
of the stored third data. 

In another embodiment of the invention, the fore- 20 
going object is attained by providing an information 
processing apparatus comprising: 

input means for inputting first data; and 
means for storing predetermined third data and 
the first data input by said input means so as to as- 25 
sociate the predetermined third data with the first 
data when second data associated with the first data 
input by said input means is included in a category of 
the predetermined third data, 

wherein the first data is searched on the basis 30 
of the stored third data. 

In another embodiment of the invention, the fore- 
going object is attained by providing an information 
processing apparatus comprising: 

input means for inputting first data; and 35 
means for storing predetermined fourth data 
associated with predetermined third data and the first 
data input by said input means so as to associate the 
predetermined fourth data with the first data when 
second data associated with the first data input by 40 
said input means is identical to the predetermined 
third data, 

wherein the first data is searched on the basis 
of the stored fourth data. 

In another embodiment of the invention, the fore- 45 
going object is attained by providing an information 
processing apparatus comprising: 

input means for inputting first data; and 

means for storing predetermined fourth data 
associated with predetermined third data and the first 50 
data input by said input means so as to associate the 
predetermined fourth data with the first data when 
second data associated with the first data input by 
said input means is included in a category of the pre- 
determined third data, 55 

wherein the first data is searched on the basis 
of the stored fourth data. 

According to another embodiment, the foregoing 



object is attained by providing an information equip- 
ment including input means for inputting information, 
storage means for storing the input information, and 
management means for selecting and processing the 
information to add associated information, the infor- 
mation equipment processing a second type of infor- 
mation having a first type of information as associat- 
ed information and a third type of information having 
the first type of information as associated informa- 
tion, wherein said management means comprises 
comparison means for comparing the first type of in- 
formation A associated with the third type of informa- 
tion with the first type of information B associated 
with the second type of information, and when a com- 
parison result from said comparison means repre- 
sents that the information A is identical to the informa- 
tion B or the information A is included in the informa- 
tion B, the second type of information is added to the 
associated information of the third type of informa- 
tion. 

Other features and advantages of the present in- 
vention will be apparent from the following descrip- 
tion taken in conjunction with the accompanying 
drawings, in which like reference characters desig- 
nate the same or similar parts throughput the figures 
thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The accompanying drawings, which are incorpo- 
rated in and constitute a part of the specification, il- 
lustrate embodiment of the invention and, together 
with the description, serve to explain the principles of 
the invention. 

Fig. 1 is a perspective view showing an overall 

apparatus according to an embodiment; 

Fig. 2 is a sectional view of the apparatus shown 

in Fig. 1; 

Fig. 3 is a view showing a display example of a 
liquid crystal display device of the embodiment 
shown in Fig. 1; 

Fig. 4A is a view for explaining a schedule data 
structure of the embodiment shown in Fig. 1; 
Fig. 4B is a view for explaining a schedule data 
structure of the embodiment shown in Fig. 1; 
Fig. 4C is a view for explaining a schedule data 
structure of the embodiment shown in Fig. 1; 
Fig. 5A is a view showing a display exam pie of the 
liquid crystal display device of the embodiment 
shown in Fig. 1; 

Fig. 5B is a view showing a display example of the 
liquid crystal display device of the embodiment 
shown in Fig. 1; 

Fig. 6 A is a view showing a display example of the 
liquid crystal display device of the embodiment 
shown in Fig. 1; 

Fig. 6B is a view showing a display example of the 
liquid crystal display device of the embodiment 
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shown in Fig. 1; 

Fig. 6C is a view showing a display example of the 
liquid crystal display device of the embodiment 
shown in Fig. 1 ; 

Fig. 7A is a diagram showing the arrangement of 5 
a comparison unit for adding information of the 
embodiment shown in Fig. 1; 
Fig. 7B is a flow chart showing the operation al- 
gorithm of the comparison unit shown in Fig. 7A; 
Fig. 8A is a view for explaining an image data 10 
structure of the embodiment shown in Fig. 1; 
Fig. 8B is a view for explaining an image data 
structure of the embodiment shown in Fig. 1; 
Fig. 9Aisa view showing a display example of the 
liquid crystal display device of the embodiment in 15 
Fig. 1; 

Fig. 9B is a view showing a display example of the 
liquid crystal display device of the embodiment in 
Fig. 1; 

Fig. 9C is a view showing a display example of the 20 
liquid crystal display device of the embodiment in 
Fig. 1; 

Fig. 9D is a view showing a display example of the 
liquid crystal display device of the embodiment in 
Fig. 1; 25 
Fig. 10 is a block diagram showing a circuit for 
controlling the apparatus in Fig. 1; 
Fig. 11 A is a view showing underlined data in the 
associted data display screen shown in Fig. 5A; 
Fig. 11 B is a view showing an image data related 30 
to "Inoue"; and 

Fig. 12 is a view showing another display exam- 
ple of Fig. 9B. 

DESCRIPTION OF THE PREFERRED 35 
EMBODIMENT 

Preferred embodiments of the present invention 
will be described in detail in accordance with the ac- 
companying drawings. 40 

Fig. 1 is a perspective view showing the overall 
apparatus of an embodiment, Fig. 2 is a sectional 
view thereof, and Fig. 10 is a block diagram of an elec- 
tric circuit. Reference numeral 1 denotes a notebook 
type electronic camera apparatus of an embodiment. 45 
The camera apparatus 1 comprises a camera body 3 
and an upper lid 5. The camera body 3 is mechanically 
and electrically connected to the upper lid 5 through 
a hinge 4 by means of a known mechanism. A switch 
(not shown) is arranged in the hinge 4. The so 
open/closed state of the upper lid 5 can be detected 
by the hinge 4. Various kinds of cards can be mounted 
in a card slot 9. New functions can be added to the 
camera body 3 in accordance with the contents of 
ROMs incorporated in the respective cards. That is, 55 
if a memory card MC incorporating an image record- 
ing memory and a camera control CPU is mounted in 
the card slot 9, the camera body 3 serves as an elec- 



tronic camera. 

The functions of the camera body in which the 
memory card MC is mounted will be described below. 

The interior of the camera body 3 is arranged, as 
shown in Fig. 2. Reference numeral 2 denotes a pho- 
tographic unit The photographic unit 2 comprises a 
CCD 22 for photoelectrically converting an object im- 
age focused by a photographic lens 21 , a diaphragm 
shutter 25, alow-pass filter 28, and an infrared outfit- 
ter 30. Reference numeral 31 denotes a focus control 
knob formed on a memberfor holding the photograph- 
ic lens. When the focus control knob is vertically slid 
in Fig. 2, a distance between the photographic fens 21 
and the CCD 22 is changed to perform focus control. 
The photographic unit is a known means, and a de- 
tailed description thereof will be omitted. 

A signal extracted as an electrical signal by the 
photographic unit 2 is sampled by a known signal 
processing circuit 40 to obtain a video signal dis- 
played on a liquid crystal display device 7 or data re- 
corded in a memory 51 in the memory card MC. The 
signal processing circuit 40 also performs signal proc- 
essing, e.g., processing for reproducing a video signal 
from the memory information to display video infor- 
mation recorded in the memory 51 on the liquid crys- 
tal display device 7. 

The liquid crystal display device 7 displays vari- 
ous kinds of information. When the camera body 
serves as an electronic still camera, the liquid crystal 
display device 7 serves as an electronic viewfinder 
(EVF) as shown in Figs. 6A, 6B, and 6C to display a 
currently photographed image. A known electrostatic 
switch 44 is arranged on the dispiay screen of the liq- 
uid crystal display device. When a pen (not shown) is 
brought into contact with the screen, the contact pos- 
ition on the screen is detected. 

Reference numeral 15 denotes a release button 
for generating a photography start trigger; and 18, a 
power switch for ON/OFF-controlling the power sup- 
ply of the camera body. A panel switch group 1 7 (Fig. 
1) is arranged on the inner surface of the upper lid 5 
and can be operated while the hinge 4 is set in an 
open state. Part of the panel switch group 17 is used 
for an electronic notebook function incorporated in a 
ROM 41 of the camera body, and the remaining part 
of the panel switch group 17 has functions changed 
depending on the types of cards mounted in the card 
slot 9. 

The camera body 3 has a CPU 45, while the 
memory card MC has a CPU 50. 

The CPU 50 controls the photographic unit 2, the 
signal processing circuit 40, the memory 51, and the 
like and performs the above camera operations. 

The operations of the release button 15, the pow- 
er switch 18, and the panel switch group 17 are de- 
tected by the CPU 45 in the camera body, and detec- 
tion signals are transmitted to the CPU 50 in the 
memory card MC. 



4 



7 



EP0 678 816 A2 



8 



A built-in clock 43 for managing time is arranged 
in the camera body 3. As electronic notebook func- 
tions, a schedule function and a data management 
function for a telephone number, an address, and the 
like are prepared in the ROM 41 of the camera body. 5 
These programs are executed by the CPU 45 in the 
camera body 3. 

The schedule and data (e.g., a telephone number 
and an address) are stored in a data memory 42 in the 
camera body 3. 10 

Functions such as the schedule function which 
are associated with the present invention will be de- 
scribed below. Other functions can be constituted by 
known techniques, and a detailed description thereof 
will be omitted. 15 

The operation of the schedule function of the ap- 
paratus of this embodiment will be described below. 
Fig. 3 shows the display screen of a schedule table of 
the liquid crystal display device 7 of this embodiment 
with reference to an example on January 22, 1 993. 1 n 20 
this display, a golf is scheduled from 8:00 to 1 7:00 on 
January 22, 1993, and a party is scheduled from 
1 7:00 to 1 9:00. The current time obtained by the built- 
in clock is always displayed in an upper portion 7a of 
the liquid crystal display device 7. 25 

Figs. 4A, 4B, and 4C are views for explaining 
schedule data. Fig. 4A shows the data structure of the 
schedule data. A schedule [n] consists of data m rep- 
resenting the number of included data and data (data 
type and contents). Fig. 4C shows an example of the 30 
contents of the schedule data shown in Fig. 3. Schedule 
[1] has seven data as associated data, i.e., event data 
"golf ', time data "1 993/01/22/08/00 -1 993/01/22/1 7/00°, 
person data "Yoshida", "Inoue", and "Goto", place 
data "ABC Gold Club", and other data "appointment: 35 
6 o'clock, Shibuya". This data structure is illustrated 
in Fig. 4B. Similarly, as shown in Fig. 4B, schedule [2] 
has the following associated data: event data "party", 
time data "1993/01/22/17/00 -1993/01/22/19/00", 
person data "Yoshida", "Inoue", "Goto", "Arai", "Yama- 40 
moto", "Hamada", and "Aoyama", and place data 
"ABC Golf Club". 

Numbers (data types) "0", "1", "2", "3", "4" follow- 
ing the above data represent the following data types 
respectively representing an event, time, a person, a 45 
place, and other data. When an input pen (not shown) 
is brought into contact with (to be referred to as "is op- 
erated at" hereinafter) a portion "golf on the display 
screen in Fig. 3, the associated data is displayed, as 
shown in Fig. 5A. When the input pen is operated at so 
a portion "party" on the screen, the associated data 
is displayed, as shown in Fig. 5B. 

The above schedule function is executed by the 
CPU 45 in the camera body 3. 

The operation of the camera function of the ap- 55 
paratus of this embodiment will be described below. 

Fig. 6A shows the display screen of the liquid 
crystal display device 7 when the camera function is 



used. The liquid crystal display device 7 serves as an 
electronic viewfinder (EVF) for displaying images 
continuously photographed by an image pickup de- 
vice. In addition, the current time counted by the built- 
in clock is displayed in the upper portion 7a of the liq- 
uid crystal display device 7. When a release switch is 
depressed in this state, the corresponding image is 
stored and recorded in the memory of the memory 
card MC. 

At this time, the photographic time, i.e., 13:28, 
January 22, 1993 at which photography is performed 
as shown in Fig. 6B is automatically read by the CPU 
45, transmitted to the CPU 50, and automatically re- 
corded as associated data. 

Schedule information is referred to to determine 
whether event data can be added. Fig. 7A is a view 
showing the function of a comparison unit C arranged 
in the CPU 45 of the camera body 3. The comparison 
unit C receives information A and information B and 
outputs the inclusion relation of these pieces of infor- 
mation. More specifically, when the contents of the 
information A are identical to those of the information 

B, the comparison unit C outputs "1". When the con- 
tents of the information A are included in those of the 
information B, the comparison unit C outputs "2"; 
otherwise, the comparison unit C outputs "3". Proc- 
essing associated with addition of event data will be 
described with reference to a flow in Fig. 7B. 

In step Si, photographic time (in this case, 13:28, 
January 22, 1993) is input to the information A (i.e., 
information A = "1993/01/22/13/28") . In step S2, n = 
1 is set In step S3, schedule time data is input to the 
information B. In step S4, the information A is com- 
pared with the information B by the comparison unit 

C. In schedule [1] shown in Fig. 4B, the information B 
is "1993/01/22/08/00 -1993/01/22/17/00", so that the 
information A is included in the information B. For this 
reason, the comparison result becomes "2". In step 
S5, an event "golf is added as data associated with 
the image. In schedule [2] in Fig. 4B, since the infor- 
mation B is "1993/01/22/17/00 - 1993/01/22/19/00", 
the information A is not associated with the informa- 
tion B. For this reason, the comparison result be- 
comes "3". In this case, the event is not added. In 
steps S6 and S7, loop control is performed to com- 
pare all the schedule data, and events are appro- 
priately added. 

The image data received by the apparatus 1 of 
this embodiment and the above additional informa- 
tion are stored in the memory 51 or the data memory 
42. The stored data are referred to by a processing 
method to be described below. 

The flow shown in Fig. 7B is executed by the CPU 
45 in the camera body 3. The event information addi- 
tion (S5) is performed such that event information to 
be added is transmitted from the CPU 45 to the CPU 
50 in the memory card MC and the CPU 50 adds the 
event data to the corresponding image data and re- 
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cords the resultant data. 

For example, when the input pen (not shown) is 
operated at a display screen portion "person" in Fig. 
6B and a name of person (in this case, "Yoshida") is 
input, it can be recorded as associated data, as shown 5 
in Fig. 6C. When the input pen is operated at a display 
screen portion "place", and a name of place (in this 
case, "ABC Golf Club") is input, it can be recorded as 
associated data, as shown in Fig. 6C. In addition, 
when the input pen is operated at the "other data", and 10 
data is input, it can be recorded as associated data. 

These processes are performed such that the 
CPU 45 in the camera body 3 detects the pres- 
ence/absence of an operation, recognizes informa- 
tion to be added, and transmits it to the CPU 50 in the 15 
memory card MC,and the CPU 50 adds the additional 
data to the corresponding image data. 

In the data structure of the above photographed 
image, as shown in Fig. 8A, an image [n] comprises 
data m representing the number of included data, 20 
data (data type and contents), and image data. An im- 
age shown in Fig. 6C has the following associated 
data: event data "golf, time data "1 993/01/22/1 3/28", 
person data 'Yoshida", and place data "ABC Golf 
Club". The data of image [1] can be illustrated, as 25 
shown in Fig. 8B. 

A search process for image data with additional 
information generated by the apparatus of this em- 
bodiment will be described below. 

Fig. HAshows underlined data in the associated 30 
data display screen shown in Fig. 5A. The underlined 
data (e.g., "Inoue") indicates the presence of image 
data having data having the same contents (i.e., data 
"Inoue") as those of the underlined data as the asso- 
ciated data. For this reason, for example, when a por- 35 
tion "Inoue" is operated, the latest data is selected 
from the image data having "Inoue" as the associated 
data and is displayed (Fig. 11 B). In addition, a search 
operation along the time axis can be performed using 
arrow keys in an upper right portion 7b. 40 

When a command for displaying schedule data of 
January 22, 1993 is input from the panel switch group 
17, the above image data with the additional informa- 
tion is searched, and a corresponding schedule table 
is displayed. As shown in Fig. 9A, to check a schedule 45 
screen display as a past schedule display, i.e., to 
check a schedule table of January 22 on January 24, 
an "image search" key is displayed in an upper right 
portion 7b of the liquid crystal display device 7. When 
this key is operated, a display shown in Fig. 9B ap- 50 
pears. In this display, there are underlined data (e.g., 
"golf). This indicates the presence of image data hav- 
ing time data included in the time data of the data 
"golf. For example, assume that a screen portion 
"golf is operated. Since the data "golf has time data 55 
"1993/01/22/08/00 -1993/01/22/17/00" as associated 
data, images having time data included in the time 
data of the data "golf are searched and displayed 



from an image closest to "1993/01/22/08/00" (Fig. 
9D). At this time, a search operation along the time 
axis can be performed using arrow keys in the upper 
right portion 7b. For example, if a photographer mem- 
orizes the date at which he played golf, an image can 
be efficiently searched by the above method. A meth- 
od of searching an image when a photographer for- 
gets the date at which he played golf will be described 
below. In the display state of Fig. 9B, when the pho- 
tographer operates the "event" key in the upper right 
portion 7b, event information stored as associated 
data of the image data in the memory card MC is dis- 
played (Fig. 9C). When the event data "golf is oper- 
ated, images having the data "golf as associated in- 
formation are searched and displayed from the latest 
one (Fig. 9D). At this time, a search operation along 
the time axis can also be performed using the arrow 
keys in the upper right portion 7b. 

In the display state shown in Fig. 9B, when one 
of the "person" key, "place" key, and "other data" key 
in the upper right portion 7b is operated, a list of per- 
son information, place information, and other infor- 
mation stored as the associated data of the image 
data in the memory card MC are displayed in accor- 
dance with the operated search key. When given in- 
formation "X" is operated in the list, images having the 
information "X" as the associated information are 
searched and displayed from the latest one. 

These processes are also executed by the CPU 
45 in the camera body 3. However, the CPU 50 per- 
forms read access to data stored in the memory card 
MC and sends necessary data to the signal process- 
ing circuit 40 and the CPU 45 in the camera body 3. 

The display screen in Fig. 9B may be displayed, 
as shown in Fig. 12. When the "person" key in the up- 
per right portion 7b is operated in Fig. 12, data indi- 
cating the presence of image data having the same 
person data as that having this data (e.g., "golf) is un- 
derlined and displayed (in this case, "party"). When 
a portion "golf is operated, images having, as the as- 
sociated data, the person data "Yoshida", "Inoue", 
and "Goto" having data "golf are searched and dis- 
played from the one closer to the time 
"1993/01/22/08/00 - 1993/01/22/17/00" (Fig. 9D). At 
this time, a search operation can be performed along 
the time axis using the arrow keys in the upper right 
portion 7b. 

In this manner, when the "person" key is operated 
as the search data, and data is input as "golf, person 
data having the data "golf as the associated data can 
be referred to and searched. For this reason, the 
search condition is an AND product of "golf and "per- 
son", thereby efficiently searching the data. In this 
case, the search operation can be performed without 
memorizing a name of person. 

The same operation as described above is per- 
formed in the state of Fig. 12 upon operations of the 
"location" key and "other data" key in the upper right 
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portion 7b. 

Black marks 7d along a time axis 7c indicate 
schedule data representing periods during which im- 
age photography has been performed. When each 
mark 7d is operated, the corresponding images can 
be searched and displayed. Therefore, when a por- 
tion of about "1 3:00" in the black mark in Fig. 1 2 is op- 
erated, the image in Fig. 9D is searched. 

Note that a method of performing a search oper- 
ation using the "event", "person", "place", or "other 
data" key can be practiced without the schedule func- 
tion (i.e., the function for displaying time and events 
in a table). 

The image data and the corresponding additional 
information which are generated by the apparatus of 
this embodiment are recorded in the memory of the 
memory card MC. This serves as a very effective 
means when the memory card MC is subjected to a 
search operation by another search unit or the image 
data in the memory card MC are to be copied in a 
large-capacity recording unit and searched. 

In the even information addition process shown in 
Fig. 7B, the information A is photographic time (date). 
However, the information A is not limited to this. For 
example, any information associated with a photo- 
graphed image, such as a name of photographer and 
a name of photographic place may be used. If the in- 
formation A is a name of photographer, a group name 
to which a plurality of photographers belong is as- 
signed to the information B. If a name of photographer 
is included in the category of a predetermined group 
name (in the process of the comparison unit in Fig. 
7A), this group name is added to the photographed 
image data and is stored. If the information A is a pho- 
tographic place, a predetermined wider area to which 
a plurality of photographic places is assigned to the 
information B. If a photographic place is included in 
the category of the predetermined area (in the proc- 
ess of the comparison unit in Fig. 7A), this area name 
is added to the photographed image data and stored. 

As has been described above, according to the 
present invention, an information equipment for proc- 
essing a second type of information having a first 
type of information as associated information, and a 
third type of information having the first type of infor- 
mation as the associated information comprises a 
comparison means for comparing the first type of in- 
formation A associated with the third type of informa- 
tion with the first type of information B associated 
with the second type of information. If a comparison 
result from the comparison unit represents that the in- 
formation A is identical to the information B or the in- 
formation A is included in the information B, the sec- 
ond type of information is added to the associated in- 
formation of the third type of information. During pho- 
tography, information for searching a photographed 
image can be automatically added, and therefore the 
recorded photographed image can be easily search- 



ed. 

Assume photography is performed at given time 
in an information equipment for processing schedule 
information having time data as associated data and 

5 image information having photographic time as asso- 
ciated data. In this case, if schedule information rep- 
resenting the same time as the given time or including 
the given time is present, this schedule data is added 
to the photographed image information as associated 

10 data, so that the photographic contents can be lab- 
eled simultaneously with photography, thereby facili- 
tating the subsequent image search operation. 

The above embodiment of the present invention 
has exemplified a notebook type electronic camera 

15 which integrally includes a photographic function, a 
schedule function, a search function, and a reproduc- 
tion function. However, these functions need not be 
integrally included to obtain the same effect as descri- 
bed above if a combination of these functions falls 

20 within the scope of the appended claims. 

As many apparently widely different embodi- 
ments of the present invention can be made without 
departing from the spirit and scope thereof, it is to be 
understood that the invention is not limited to the spe- 

25 cific embodiments thereof except as defined in the 
appended claims. 



Claims 

30 

1. An information processing method comprising: 

an input step of inputting first data; and 
a step of storing predetermined third data 
and the first data input in the input step so as to 
35 associate predetermined third data with the first 
data when second data associated with the first 
data input in the input step is identical to the pre- 
determined third data, 

wherein the first data is searched on the 
40 basis of the stored third data. 

2, The method according to claim 1, wherein the 
first data is image data. 

45 3. The method according to claim 2, wherein the 
second data represents time at which the first 
data is input in the input step. 

4. The method according to claim 2, wherein the 
so second data represents a name of person who in- 
puts the first data in the input step. 

5. The method according to claim 2, wherein the 
second data represents a place at which the first 

55 data is input in the input step. 

6. An information processing method comprising: 

an input step of inputting first data; and 
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a step of storing predetermined third data 
and the first data input in the input step so as to 
associate the predetermined third data with the 
first data when second data associated with the 
first data input in the input step is included in a 
category of the predetermined third data, 

wherein the first data is searched on the 
basis of the stored third data. 

7. The method according to claim 6, wherein the 
first data is image data. 

8. The method according to claim 6, wherein the 
second data represents time at which the first 
data is input in the input step, and 

the third data represents a predetermined 
time interval. 

9. The method according to claim 6, wherein the 
second data represents a name of person who in- 
puts the first data in the input step, and 

the third data represents a name of group 
to which a predetermined name of person be- 
longs. 

10. The method according to claim 6, wherein the 
second data represents a place at which the first 
data is input in the input step, and 

the third data represents a name of area to 
which a predetermined place belongs. 

11. An information processing method comprising: 

an input step of inputting first data; and 
a step of storing predetermined fourth data 
associated with predetermined third data and the 
first data input in the input step so as to associate 
the predetermined fourth data with the first data 
when second data associated with the first data 
input in the input step is identical to the predeter- 
mined third data, 

wherein the first data is searched on the 
basis of the stored fourth data. 

12. The method according to claim 11, wherein the 
first data is image data. 

13. The method according to claim 12, wherein the 
second data represents time at which the first 
data is input in the input step. 

14. The method according to claim 12, wherein the 
second data represents a name of person who in- 
puts the first data in the input step. 

15. The method according to claim 12, wherein the 
second data represents a place at which the first 
data is input in the input step. 



16. The method according to claim 12, wherein the 
fourth data is event data associated with the third 
data. 

5 17. An information processing method comprising: 
an input step of inputting first data; and 
a step of storing predetermined fourth data 
associated with predetermined third data and the 
first data input in the input step so as to associate 
10 the predetermined fourth data with the first data 
when second data associated with the first data 
input in the input step is included in a category of 
the predetermined third data, 

wherein the first data is searched on the 
15 basis of the stored fourth data. 

18, The method according to claim 17, wherein the 
first data is image data. 

20 19. The method according to claim 17, wherein the 
second data represents time at which the first 
data is input in the input step, and 

the third data represents a predetermined 
time interval. 

25 

20. The method according to claim 17, wherein the 
second data represents a name of person who in- 
puts the first data in the input step, and 

the third data represents a name of group 
30 to which a predetermined name of person be- 
longs. 

21. The method according to claim 17, wherein the 
second data represents a place at which the first 

35 data is input in the input step, and 

the third data represents a name of area to 
which a predetermined place belongs. 

22. The method according to claim 17, wherein the 
40 fourth data is event data associated with the third 

data. 

23. An information processing apparatus comprising: 

input means for inputting first data; and 
45 means for storing predetermined third data 

and the first data input by said input means so as 
to associate the predetermined third data with the 
first data when second data associated with the 
first data input by said input means is identical to 
so the predetermined third data, 

wherein the first data is searched on the 
basis of the stored third data. 

24. The apparatus according to claim 23, wherein the 
55 first data is image data. 

25. The apparatus according to claim 23, wherein the 
second data represents time at which the first 
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data is input by said input means. 

26. The apparatus according to claim 23, wherein the 
second data represents a name of person who in- 
puts the first data by said input means. 

27. The apparatus according to claim 23, wherein the 
second data represents a place at which the first 
data is input by said input means. 

28. An information processing apparatus comprising: 

input means for inputting first data; and 
means for storing predetermined third data 
and the first data input by said input means so as 
to associate the predetermined third data with the 
first data when second data associated with the 
first data input by said input means is included in 
a category of the predetermined third data, 

wherein the first data is searched on the 



31. The apparatus according to claim 28, wherein the 
second data represents a name of person who in- 
puts the first data by said input means, and 

the third data represents a name of group 
to which a predetermined name of person be- 
longs. 

32. The apparatus according to claim 28, wherein the 
second data represents a place at which the first 
data is input by said input means, and 

the third data represents a name of area to 
which a predetermined place belongs. 



34. The apparatus according to claim 33, wherein the 
first data is image data. 



35. The apparatus according to claim 33, wherein the 
second data represents time at which the first 
data is input by said input means. 

5 36. The apparatus according to claim 33, wherein the 
second data represents a name of person who in- 
puts the first data by said input means. 

37. The apparatus according to claim 33, wherein the 
10 second data represents a place at which the first 

data is input by said input means. 

38. The apparatus according to claim 33, wherein the 
fourth data is event data associated with the third 

15 data. 

39. An information processing apparatus comprising: 
input means for inputting first data; and 
means for storing predetermined fourth 

data associated with predetermined third data 
and the first data input by said input means so as 
to associate the predetermined fourth data with 
the first data when second data associated with 
the first data input by said input means is included 
in a category of the predetermined third data, 

wherein the first data is searched on the 
basis of the stored fourth data. 

40. The apparatus according to claim 39, wherein the 
30 first data is image data. 

41. The apparatus according to claim 39, wherein the 
second data represents time at which the first 
data is input by said input means, and 

35 the third data represents a predetermined 

time interval. 

42. The apparatus according to claim 39, wherein the 
second data represents a name of person who in- 

40 puts the first data by said input means, and 

the third data represents a name of group 
to which a predetermined name of person be- 
longs. 

43. The apparatus according to claim 39, wherein the 
second data represents a place at which the first 
data is input by said input means, and 

the third data represents a name of area to 
which a predetermined place belongs. 

44. The apparatus according to claim 39, wherein the 
fourth data is event data associated with the third 
data. 

55 45. An information equipment including input means 
for inputting information, storage means for stor- 
ing the input information, and management 
means for selecting and processing the informa- 



basis of the stored third data. 20 

29. The apparatus according to claim 28, wherein the 
first data is image data. 

30. The apparatus according to claim 28, wherein the 25 
second data represents time at which the first 
data is input by said input means, and 

the third data represents a predetermined 
time interval. 



33. An information processing apparatus comprising: 

input means for inputting first data; and 45 
means for storing predetermined fourth 
data associated with predetermined third data 
and the first data input by said input means so as 
to associate the predetermined fourth data with 
the first data when second data associated with so 
the first data input by said input means is identical 
to the predetermined third data, 

wherein the first data is searched on the 
basis of the stored fourth data. 
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tion to add associated information, the informa- 
tion equipment processing a second type of infor- 
mation having a first type of information as asso- 
ciated information and a third type of information 
having the first type of information as associated 5 
information, wherein said management means 
comprises comparison means for comparing the 
first type of information A associated with the 
third type of information with the first type of in- 
formation B associated with the second type of 10 
information, and when a comparison result from 
said comparison means represents that the infor- 
mation A is identical to the information B or the in- 
formation A is included in the information B, the 
second type of information is added to the asso- 15 
dated information of the third type of information. 

46. The equipment according to claim 45, wherein 
the first type of information is time data, and the 
third type of information is image data. 20 

47. A recorder for audio, video and/or still pictures, 
wherein index information is stored electronically 
in association with recorded images or sound, au- 
tomatically by reference to pre-stored schedule 25 
information. 

48. An apparatus according to claim 47 further com- 
prising means for searching by reference to the 
stored index information. 30 

49. Amethod or apparatus having the features of any 
combination of the preceding claims. 
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